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I.  INTRODUCTION 


Data  collection  and  availability  has  opened  up  many  opportunities  to  use  that  data 
in  a  way  that  helps  us  become  better  managers.  There  are  several  issues  surrounding  the 
use  of  data  and  specifically  metrics.  For  instance,  is  the  data  accurate?  Is  that  data  in  a 
form  that  makes  it  reasonable  to  expend  the  energy  retrieving  it?  Is  the  expended  energy 
(resources)  worth  the  information  that  is  gleaned  from  the  data? 

The  purpose  of  this  project  is  to  provide  an  evaluation  of  the  current  Department 
of  the  Navy  Procurement  performance  measurements  (metrics)  being  collected  from  the 
Major  Naval  Systems  Commands  (SYSCOMs).  Specifically,  the  metrics  are  being 
collected  from  the  Head  of  Contracting  Activities  (HCAs)  within  those  SYSCOMs.  This 
evaluation  will  hopefully  contribute  to  deciding  if  the  current  procurement  metrics  are 
useful,  appropriate,  and  of  value. 


A.  BACKGROUND 

In  fiscal  year  2002  the  SYSCOMs  were  the  subject  of  a  Booz  Allen  and  Hamilton 
effectiveness  and  efficiency  study.  The  study  resulted  in  eleven  recommendations:  nine 
for  internal  spend  and  two  for  industry  spend.  Of  these  eleven  recommendations  two 
applied  specifically  to  the  procurement  community.  Within  these  two  recommendations, 
Booz  Allen  and  Hamilton  identified  an  “enabler”  for  the  procurement  community.  The 
enabler  was  “define  and  institute  contracting  performance  measures  and  reporting” 
(Procurement  Metrics  Guide  [PMG],  vl4.3,  p.  4). 

The  office  of  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition,  ASN  (RDA),  then  tasked  the  office  of  the  Deputy  Assistant  Secretary  of  the 
Navy  for  Acquisition  Management,  DASN  ACQ,  to  “provide  a  ‘dashboard’  or 
spreadsheet  for  examining  the  effectiveness  and  efficiency  of  the  Department  of  the 
Navy’s  Material  establishment”  (PMG,  vl4.3,  p.  4).  DASN  ACQ  created  a  working 
group  consisting  of  HCAs  representatives  from  ten  SYSCOMs  to  assist  in  the 

development  of  standard  procurement  performance  measures. 
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The  working  group  decided  on  five  focus  areas  as  part  of  their  Balanced 
Scorecard1 2  approach  to  developing  the  procurement  performance  measures.  The  five 
focus  areas  are  Customer,  People,  Process,  Financial,  and  Value.  Each  area  has  a 
strategic  theme,  objectives,  and  associated  metrics.  The  dashboard  was  designed  for  use 
specifically  within  the  Navy’s  Procurement  Community,  both  Military  and  Civilian,  and 
to  serve  as  an  executive  level  summary  of  the  Navy’s  procurement  performance. 

The  metrics  put  together  by  the  working  group  measure  the  procurement  activities 
of  HCA’s  representing  the  following  ten  SYSCOMs: 


Marine  Corps  Installations  and  Logistics  (MARCOR  I&L) 
Marine  Corps  Systems  Command  (MARCORSYSCOM) 
Military  Sealift  Command  (MSC) 

Naval  Air  Systems  Command  (NAVAIR) 

Naval  Facilities  Engineering  Command  (NAVFAC) 

Naval  Sea  Systems  Command  (NAVSEA) 

Naval  Supply  Systems  Command  (NAVSUP/FISC’s/ICP’s) 
Office  of  Naval  Research  (ONR) 

Space  and  Naval  Warfare  Systems  Command  (SPAWAR) 
Strategic  Systems  Project  Office  (SSP) 


B.  PROBLEM  IDENTIFICATION 

The  HCA  working  group  developed  2 1  metrics  that  they  stated,  when  used,  would 
give  an  overall  picture  of  the  effectiveness  and  efficiency  of  the  Navy’s  procurement 
function.  The  Office  of  the  DASN  ACQ  is  currently  collecting  data  for  the  metrics  from 
the  SYSCOMs  as  well  as  pulling  data  for  the  metrics  from  existing  databases.  However, 
the  office  of  the  DASN  ACQ  is  not  receiving  all  of  the  HCA  responsible  metrics  from 
each  of  the  ten  SYSCOMs.  Some  simply  just  do  not  have  the  infonnation  needed  to 
answer  DASN’s  metrics  data  call  or  do  not  have  the  necessary  resources  in  place  to 
retrieve  the  data.  This  creates  an  issue.  The  metrics  need  to  be  evaluated  in  order  to 
determine  if  they  are  useful,  appropriate,  and  of  value.  However,  without  a  relatively 

1  The  Balanced  Scorecard  is  a  system  developed  by  Robert  S.  Kaplan  and  David  P.  Norton.  The 
scorecard  takes  an  organization’s  mission  and  strategy  and  translates  that  into  goals  and  measures,  both 
financial  and  non-financial.  The  basic  scorecard  uses  four  areas  for  measurement:  financial,  customer, 
internal  business,  and  innovation  and  learning.  The  metrics  working  group  adapted  the  basic  scorecard  to 
include  five  focus  areas  in  order  to  fit  their  specific  needs. 
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complete  set  of  metrics  it  is  difficult  to  determine  whether  or  not  the  effort  should  be 
made  to  create  and  generate  a  full  and  accurate  metric  report  from  each  of  the  ten 
SYSCOMs. 
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II.  LITERATURE  REVIEW  AND  FRAMEWORK  USED 


A.  MANAGEMENT  CONTROL  SYSTEMS 

Communication  is  a  critical  element  of  the  operations  of  organizations. 
Information  of  all  kinds  is  communicated  through  both  formal  and  infonnal  channels. 

One  reason  managers  of  organizations  are  interested  in  the  information  communicated  is 
because  it  sends  signals  describing  the  health  of  the  organization.  Communication 
channels  also  provide  a  means  for  the  managers  to  disseminate  information.  Senior 
managers  can  use  those  channels  for  communicating  a  strategic  vision,  profit  goals  or 
receiving  infonnation  about  organizational  developments  or  perfonnance. 

All  of  this  information  serves  as  one  of  the  controls  in  the  organization.  Controls 
can  be  put  in  place  through  many  methods,  varying  from  internal  control  systems  focused 
on  inventory  tracking  to  a  company’s  mission  statement.  In  Robert  Simon’s  book,  Levers 
of  Control,  he  “focuses  primarily  on  the  informational  aspects  of  management  control 
systems — the  levers  managers  use  to  transmit  and  process  information  within 
organizations”  (Simons,  1995,  p.  5).  He  uses  the  following  definition  of  management 
control  systems:  management  control  systems  are  the  formal,  information-based  routines 
and  procedures  managers  use  to  maintain  or  alter  patterns  in  organizational  activities 
(Simons,  1995,  p.  5). 

The  information-based  management  control  system  involves  information 
transmission  not  only  from  the  top  management  level  to  lower  levels,  but  also 
information  flow  in  the  opposite  direction.  The  information  flow  from  the  lower  levels  to 
top  management  allows  for  the  monitoring  of  implemented  strategies  and  other  efforts 
that  were  communicated  down  from  top  management.  Figure  1  illustrates  this  point. 


5 


Top  Management 


/ 

Information 

about 

/ 

Information 

about 

\ 

\ 

strategic 

intended 

Information 

Information 

domain 

strategics 

about 

about 

\ 

and  plans 

\ 

progress 
in  achieving 
intended 
strategies 

/' 

emerging 
threats  and 
opportunities 

/ 

Lower  Organizational  Levels 


Figure  1.  Information  Needs  of  Top  Managers  (Simons,  1995,  p.  6) 


As  stated  above,  organizational  control  can  take  many  forms.  A  well  performing 
organization  will  likely  have  some  sort  of  control  throughout  all  facets  of  their 
organization.  This  may  sound  oppressive  and  stifling  to  the  employee’s  innovation  and 
motivation.  However,  this  does  not  have  to  be  the  case.  Control  does  not  necessarily 
mean  micro-managing. 

Robert  Simons’  framework  has  business  strategy  as  its  heart.  This  is  how  Simons 
describes  the  framework: 

Business  Strategy — how  a  firm  competes  and  positions  itself  vis-a-vis  its 
competitors — is  at  the  core  of  the  analysis.  The  second  level  introduces 
four  key  constructs  that  must  be  analyzed  and  understood  for  the 
successful  implementation  of  strategy:  core  values,  risks  to  be  avoided, 
critical  performance  variables,  and  strategic  uncertainties.  Each  construct 
is  controlled  by  a  different  system,  or  lever,  the  use  of  which  has  different 
implications.  These  levers  are:  (Simons,  1995,  p.  6) 


1 .  belief  systems,  used  to  inspire  and  direct  the  search  for  new  opportunities; 

2.  boundary  systems,  used  to  set  limits  on  opportunity-seeking  behavior; 
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3.  diagnostic  control  systems,  used  to  motivate,  monitor,  and  reward 
achievement  of  specified  goals;  and 

4.  interactive  control  systems,  used  to  stimulate  organizational  learning  and  the 
emergence  of  new  ideas  and  strategies  (Simons,  1995,  p.  7) 

The  following  figure  is  the  representation  of  the  Levers  of  Control,  with  business 
strategy  at  the  core,  surrounded  by  key  constructs  to  be  analyzed  and  understood,  and 
controlled  by  the  four  systems,  or  levers. 


BELIEF 

SYSTEMS 


BOUNDARY 

SYSTEMS 


INTERACTIVE  DIAGNOSTIC 

CONTROL  CONTROL 

SYSTEMS  SYSTEMS 


Figure  2.  Levers  of  Control  Framework  (Simons,  1995,  p.  7) 


B.  BELIEF  AND  BOUNDARY  SYSTEMS 
1.  Belief  Systems 

Belief  systems,  as  defined  by  Simons,  “are  the  explicit  set  of  organizational 

definitions  that  senior  managers  communicate  formally  and  reinforce  systematically  to 
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provide  basic  values,  purpose,  and  direction  for  the  organization”  (Simons,  2000,  p.  276). 
The  belief  system  provides  the  means  of  control  over  the  organization’s  foundational 
principles  and  ideals. 

These  principles  and  ideals  are  often  articulated  by  an  organization’s  founders  and 
provide  the  basis  for  employee  action  and  motivation  when  rules  and  regulations  alone 
will  not  sufficiently  address  the  issue  or  situation  that  employees  face.  They  address  such 
things  as  responsibilities  to  customers  and  other  employees.  This  set  of  principles  is 
typically  called  a  set  of  core  values.  Simons  defines  core  values  as  “the  beliefs  that 
define  basic  principles,  purpose,  and  direction”  (Simons,  2000,  p.  276).  The  core  values 
often  take  the  form  of  a  mission  statement  or  credo. 

2.  Boundary  Systems 

Belief  systems  go  a  long  way  to  inspire  and  motivate  employees  to  achieve.  They 
give  them  a  sense  of  purpose  to  their  work.  This  is  beneficial,  but  there  is  a  limit  to  what 
employees  can  and  should  do.  The  employees  need  to  be  motivated,  innovative  and  serve 
the  customer,  but  they  should  not  pursue  an  avenue  which  opens  up  the  business  to  undue 
risk.  An  organization  needs  to  understand  what  risks  they  want  to  avoid  and  apply  a  set 
of  boundaries  to  employees  to  minimize  the  likelihood  of  that  risk  occurring.  This  lever 
of  control,  boundary  systems,  plays  an  integral  role  in  business  strategy.  Without 
boundaries  the  belief  system  is  seriously  degraded.  Enthusiasm  can  lead  to  undesired 
outcomes  if  left  unchecked.  The  boundary  system  is  like  the  brakes  on  enthusiasm  and 
innovation.  Simons  uses  the  example  of  brakes  on  a  car: 

Cars  have  brakes  for  two  reasons.  First,  and  most  obvious,  they  allow  the 
driver  to  slow  the  car  down  and  stop  safely.  However,  cars  have  brakes 
for  another  reason.  They  give  the  driver  the  confidence  to  go  very  fast. 

Imagine  a  high-perfonnance  racing  car  on  a  speedway.  The  driver  can 
operate  at  top  speeds  only  if  he  knows  that  he  can  rely  on  excellent  brakes 
to  control  the  car  on  right  turns.  Like  the  fastest  cars,  managers  of  high- 
performance  businesses  need  the  best  brakes  to  control  strategic  risks  that 
are  an  inevitable  consequence  of  driving  their  businesses  to  their 
maximum  potentials  (Simons,  2000,  p.  275). 

A  boundary  system  basically  tells  employees  what  not  to  do.  It  lets  the 
employees  know  which  actions  management  deems  unacceptable.  This  may  seem  like  a 
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hindrance  on  their  innovation,  but  in  fact,  this  is  what  allows  them  to  be  innovative.  It 
allows  them  to  be  innovative  in  a  way  which  reduces  the  likelihood  that  an  organization 
will  be  exposed  to  risk.  Another  approach  that  a  manager  could  employ  would  be  one 
telling  employees  what  to  do.  This  method  ensures  that  employees  do  not  perform 
actions  that  top  management  feels  are  risky.  This  system  also  limits  what  individuals  can 
actually  do.  Now  that  the  employees  are  told  what  to  do,  they  will  come  to  work,  follow 
procedures,  and  leave.  There  is  minimal  opportunity  for  innovation  within  this  system. 

C.  INTERACTIVE  AND  DIAGNOSTIC  CONTROL  SYSTEMS 

The  following  two  systems,  which  complete  the  levers  of  control  model,  mean 
quite  literally  what  their  titles  portray.  Managers  use  these  two  systems  in  quite  different 
ways.  Interactive  control  systems  require  a  good  deal  of  interaction  between  top 
managers  and  employees  discussing  possible  emerging  opportunities  or  threats  to  their 
business  given  the  changing  market  environment.  The  system’s  intent  is  to  use  the 
information  gained  from  these  interactions  to  modify  strategy  as  the  environment 
changes.  Diagnostic  control  systems  are  used  to  diagnose  issues  or  discrepancies  which 
vary  from  the  intended  strategy  or  expected  output  of  the  organization.  Diagnostic 
control  systems  are  used  to  monitor  performance  of  critical  perfonnance  variables  and 
monitor  the  implementation  of  the  organization’s  intended  strategy. 

1.  Interactive  Control  Systems 

Interactive  control  systems  “are  the  formal  infonnation  systems  that  managers  use 
to  personally  involve  themselves  in  the  decision  activities  of  subordinates.  Simply  stated, 
[they]  are  the  hot  buttons  for  senior  managers.  They  provide  the  infonnation  that  the 
boss  pays  a  lot  of  attention  to  and  are  used  to  create  an  ongoing  dialogue  with 
subordinates”  (Simons,  2000,  p.  216). 

Interactive  control  systems  are  centered  on  “strategic  uncertainties”  or  the 

“emerging  threats  and  opportunities  that  could  invalidate  the  assumptions  upon  which  the 

current  business  strategy  is  based”  (Simons,  2000,  p.  215).  These  uncertainties,  by 

definition,  are  not  known  beforehand,  they  emerge  over  time.  The  environment  is 

constantly  evolving  and  changing  and  then  so  too  does  the  organization  if  it  is  to  survive. 

Therefore,  business  strategy,  the  core  of  the  levers  of  control  model,  has  to  be  reviewed 
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and  examined  to  determine  if  the  organization’s  current  strategy  will  put  it  in  a  position 
to  be  successful.  Top  managers  need  to  interact  with  their  employees  to  uncover  these 
strategic  uncertainties. 

Examples  of  strategic  uncertainties  are  rapidly  changing  technologies  which  could 
possibly  render  the  organizations’  value  proposition  worthless  and  governmental  relaxing 
or  tightening  of  regulations.  In  the  first  case,  this  represents  a  threat  to  the  organization, 
but  strategic  uncertainties,  as  stated  above,  can  be  avenues  for  opportunities  for  the 
organization  as  well.  Both  could  be  true  for  the  second  situation,  changing  government 
regulations.  This  could  help  or  hurt  the  organization. 

One  way  to  identify  these  uncertainties  is  to  create  an  environment  of  open 
dialogue  throughout  the  organization  which  focuses  on  examining  and  revising  the 
business  strategy  given  the  complex  and  changing  environment.  Everyone  needs  to  be  on 
board  and  focus  on  the  questions  that  need  to  be  asked,  thereby  ensuring  a  successful 
future.  This  open-dialogue  environment  is  created  through  an  interactive  control  system. 

The  use  of  interactive  control  systems  involves  thorough  evaluation  of  internal 
reports  and  efforts  and  questioning  subordinate’s  assumptions  in  those  reports  in  an  effort 
to  challenge  old  ways  of  thinking.  This  challenge  should  spark  debate  and  dialogue  and 
ultimately  lead  to  a  pooling  of  information,  gained  throughout  the  organization,  for  top 
managers  to  pour  over  and  help  guide  the  identification  of  strategic  uncertainties.  This 
method  of  interactive  information  gathering  and  use  of  that  information  is  very  dissimilar 
to  how  a  diagnostic  control  system  is  used. 

2.  Diagnostic  Control  Systems 

This  research  project  focuses  on  the  final  lever  of  control,  the  diagnostic  control 
system.  Simons  defines  a  diagnostic  control  system  as  the  “formal  information  systems 
that  managers  use  to  monitor  organizational  outcomes  and  correct  deviations  from  preset 
standards  of  performance”  (Simons,  2000,  p.  209).  The  purpose  of  a  diagnostic  control 
system  is  to  enable  management  by  exception.  The  care  and  attention  used  to  construct 
the  system  and  the  validity  of  the  system  determines  whether  or  not  managing  by 
exception  can  be  successful.  This  system  is  only  as  good  as  the  measurements  used  to 
build  it,  as  will  be  seen  later. 
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The  basic  premise  of  a  diagnostic  control  system  is  that  of  variance  analysis.  This 
system  compares  actual  output  or  results  with  those  that  were  expected  or  set  as  goals.  If 
the  results  are  within  the  bounds  of  what  was  intended  to  be  accomplished  then  the 
manager  need  not  search  any  further  because  things  are  on  track.  If,  on  the  other  hand,  a 
result  is  off  target,  the  manager  will  note  the  variance  and  subsequently  investigate 
possible  reasons  why  the  deviations  occurred.  The  process  is  referred  to  as  feedback. 

This  is  equivalent  to  a  “cybernetic  feedback  model,”  shown  in  Figure  3.  In  such  a  model, 
every  measure  has  a  benchmark  or  standard  that  represents  the  expected  output.  The 
deviation  that  is  the  result  of  comparing  the  output  to  the  standard  provides  feedback  to 
the  input  or  process  parts  of  the  model.  The  purpose  of  the  feedback  is  to  ensure  that 
performance  targets  are  met  in  the  future  by  adjusting  the  inputs. 


Standard 


Inputs 


Process 
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Outputs 


Feedback 


Figure  3.  Cybernetic  Feedback  Model  (Simons,  2000,  p.  61) 

A  diagnostic  control  system  is  applicable  in  many  situations.  These  situations 
include  using  tools  such  as  balanced  scorecards,  budgets,  profit  plans  and  standard  cost¬ 
accounting  systems  in  a  diagnostic  manner.  Simons  argues  that  there  are  two  key  reasons 
for  a  using  diagnostic  control  system:  “to  implement  strategy  effectively  and  conserve 
scarce  management  attention”  (Simons,  2000,  p.  209). 

Time  is  scarce  for  managers.  There  are  thousands  of  tasks  and  limited  hours  in 
the  day.  The  goal  of  a  diagnostic  control  system  is  to  encapsulate  relevant  variables  into 
a  user  friendly  fonnat  that  managers  can  use  in  a  time-efficient  manner  to  give  them  a 
snapshot  of  the  health  of  their  organization.  Simons  uses  the  example  of  the  speedometer 
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in  your  car.  During  a  long  trip  it  would  be  incredibly  tiresome  to  be  constantly 
monitoring  the  speedometer  and  making  small  adjustments  to  your  speed.  All  of  the  time 
and  energy  would  be  used  doing  this  task.  Instead,  the  car  can  be  put  on  cruise  control 
which  frees  the  time  to  focus  on  other  things.  This  is  in  essence  what  a  diagnostic  control 
system  will  do.  The  parameters  are  set  and  only  when  the  output  falls  out  of  range  is 
there  a  signal.  When  that  occurs,  then  focus  should  be  directed  to  get  the  output  back 
within  bounds. 

The  particular  variables  chosen  to  be  measured  must  be  carefully  selected. 

Simons  calls  these  variables,  critical  performance  variables.  He  defines  them  as  “those 
factors  that  must  be  achieved  or  implemented  successfully  for  the  intended  strategy  of  the 
business  to  succeed”  (Simons,  2000,  p.  209).  The  diagnostic  control  system  is  in  place  to 
monitor  whether  the  strategy,  through  these  critical  performance  variables,  is  being 
implemented.  This  system  links  the  strategy  to  the  measurable  perfonnance  variables. 
The  diagnostic  control  system  is  crucial  to  communication  and  effectively  implementing 
strategy  in  large  organizations. 

3.  Risks  in  Diagnostic  Control  Systems 

The  hands  off  nature  of  a  diagnostic  control  system,  after  extensive  attention  and 
work  to  ensure  a  properly  constructed  and  aligned  system,  is  a  very  desirable  attribute. 
However,  this  system  is  not  without  its  risks.  Simons  identifies  three  risks  associated 
with  using  a  diagnostic  control  system;  “measuring  the  wrong  variables,  building  slack 
into  targets,  and  gaming  the  system”  (Simons,  2000,  pp.  212-213). 

Measuring  the  Wrong  Variables  People  pay  attention  to  things  upon  which  they 
are  evaluated.  If  senior  management  identifies  incorrect  or  misaligned  variables  to  be 
measured,  then  subordinates  will  be  concentrating  on  achieving  goals  that  do  not 
significantly  contribute  to  the  overall  intended  business  strategy.  Measuring  the  wrong 
variables  could  hinder  or  derail  strategy  implementation. 

Building  Slack  into  Targets  If  employees’  performance  is  evaluated  on  the 
achievement  of  a  goal,  then  their  natural  tendency  will  be  to  increase  the  likelihood  that 
they  achieve  that  goal.  This  can  be  achieved  in  one  of  two  ways:  (1)  through  hard  work 
or  (2)  by  setting  easily  attainable  goals.  Managers  need  to  be  cognizant  of  the  latter 

behavior  and  ensure  that  the  goals  that  are  set  are  challenging  ones. 
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Gaming  the  System  In  general,  employees  will  work  hard  towards 
accomplishing  goals  upon  which  they  are  measured.  This  hard  work  can  be  focused 
away  from  achieving  the  desired  goal  in  the  manner  envisioned  by  top  management. 

“This  misdirected  effort  is  called  gaming”  (Simons,  2000,  p.  213).  Simons  (2000) 
identifies  three  common  gaming  activities:  smoothing,  biasing,  and  illegal  acts.  The  act 
of  “smoothing”  attempts  to  show  better  performance  by  manipulating  the  timing  and 
recording  of  transactions.  An  example  of  this  might  occur  if  the  goals  for  small  business 
contract  awards  (in  a  given  fiscal  year)  are  reached.  Rather  than  make  the  current  year 
look  better  by  recording  additional  small  business  contract  awards,  after  the  goal  is  met, 
the  employee  may  chose  to  record  any  additional  contract  awards  towards  next  year’s 
goals.  “Biasing”  occurs  when  managers  chose  to  report  goals  that  have  been  achieved, 
but  downplay  or  bury  bad  news  or  goals  that  were  not  achieved.  The  attempt  is  to  control 
and  bias  the  information  flow.  Finally,  pressure  to  achieve  can  sometimes  prove  to  be 
too  much.  Individuals  may  commit  “illegal  acts”  in  order  to  meet  their  goals  and  earn  a 
bonus.  These  acts  might  violate  the  law  or  company  policy  and  therefore  would  be 
considered  illegal. 

D.  SELECTING  PERFORMANCE  MEASURES/METRICS 

In  order  to  mitigate  these  risks,  it  is  imperative  that  the  correct  perfonnance 
measures  or  metrics  are  chosen.  Performance  measures  may  seem  to  be  intuitive  and 
rudimentary  in  nature,  but  the  opposite  is  true  in  many  situations.  The  performance 
measures  selected  have  profound  meaning  not  only  for  the  managers  who  will  be  using 
them  to  make  decisions,  but  more  importantly,  for  those  who  are  being  measured. 

Robert  Simons  identifies  three  tests  which  a  perfonnance  measure  must  be 
subjected  to  in  order  to  detennine  if  the  measure  is  a  suitable  one:  (1)  Does  the  measure 
align  with  strategy?  (2)  Can  it  be  measured  effectively?  (3)  Is  it  linked  to  value?  (Simons, 
2000,  p.  234) 

Test  one  is  whether  a  measure  is  aligned  with  strategy.  When  a  metric  or  measure 
is  created,  the  managers  creating  it  must  know  that  by  doing  so  they  are  telling  their 
employees  what  is  important.  Why  have  the  measure  if  it  is  not  meaningful?  Thus,  a 
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measure  must  be  in  congruence  with  the  strategy  of  the  organization.  This  not  only 
reinforces  the  organization’s  strategy  with  employees,  but  it  inherently  supports  the 
achievement  of  the  strategy. 

Just  as  important  as  its  alignment  with  strategy,  a  metric  must  be  able  to  be 
measured  effectively,  Test  2.  There  is  little  or  no  value  to  having  a  metric  that  is 
supposed  to  be  telling  managers  one  thing,  but  is  giving  them  something  different.  In 
reality  the  metric  might  either  be  too  difficult  or  complex  to  measure  or  simply  the  wrong 
metric.  In  the  later  case,  the  correct  metric  can  replace  the  incorrect  one,  but  the  former 
case  is  a  more  difficult  one.  It  requires  a  more  in-depth  analysis  of  the  strategy  and  a 
determination  if  the  ability  to  capture  the  essence  of  the  measure  exists. 

In  Robert  Simons’  discussion  of  Test  2,  can  it  be  measured  effectively,  he 
mentions  three  distinct  adjectives  against  which  measures  can  be  evaluated.  He  states, 
“Ideally,  measures  should  be  objective,  complete,  and  responsive.”  (Simons,  2000,  p. 

235)  These  are  referred  to  as  the  “Nature  of  Measures.” 

Managers  must  have  solid  information  on  which  to  base  their  decisions.  Solid 
information  comes  from  having  a  clear  formula  for  the  metric  and  unambiguous  inputs  to 
the  formula.  When  the  metric  is  objective  it  can  be  independently  verified.2  If  the  metric 
is  a  subjective  one,  then  Simons  goes  on  to  say  in  that  situation  “trust  must  be  high, 
because  the  subordinate  must  have  confidence  that  the  subjective  judgment  is  fair  and 
will  be  used  appropriately.”  (Simons,  2000,  p.  236) 

Completeness  and  responsiveness  are  not  as  clearly  ascertained.  Metrics  can  have 
varying  degrees  of  each.  A  metric  is  an  attempt  to  explain  an  aspect  of  the  organization. 
How  well  the  metric  explains  that  particular  aspect  of  the  organization  is  its  degree  of 
completeness.  When  a  metric  is  complete  it  captures  all  of  the  relevant  attributes  of 
success.  A  responsive  metric  is  one  that  responds  to  the  actions  of  managers.  The 
process  of  feedback  and  variance  analysis  used  in  a  diagnostic  control  system  relies 
partly  on  an  organization’s  ability  to  make  corrections  to  inputs  to  decrease  the  variance 
between  the  output  and  the  perfonnance  standard.  If  the  metric  is  not  responsive  then 

2  An  objective  metric  is  not  the  only  type  of  measure  that  is  acceptable.  Although  a  subjective  metric 
may  not  be  independently  verified  to  the  same  extent  as  an  objective  one,  a  subjective  metric  can  be  very 
valuable  in  some  cases. 
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managers  can  not  influence  the  output.  This  may  have  negative  impacts  on  morale 
because  it  can  create  a  situation  where  managers  are  being  evaluated  on  their  ability  to 
meet  perfonnance  targets,  but  do  not  have  the  ability  to  directly  influence  the  results. 

A  classic  example  which  illustrates  responsiveness  and  completeness  is  the  “sales 
calls”  example.  A  manager  chooses  to  evaluate  employee  sales  performance  by 
measuring  the  number  of  sales  calls  made  each  week.  The  metric  is  responsive  because 
the  employee  has  influence  on  it.  While  this  can  be  good,  it  can  also  create  risk.  So,  if 
employees  know  they  are  being  evaluated  on  the  number  of  sales  calls  per  week  then  they 
can  easily  increase  the  number  of  sales  calls  without  regard  for  making  an  actual  sale  at 
each  stop.  Because  this  metric  fails  to  capture  the  key  factor  of  actual  sales  it  fails  the 
completeness  test. 

Typically,  lower-level  jobs  which  require  more  routine  work  can  achieve  metrics 
that  are  objective,  complete  and  responsive  in  nature.  The  higher  up  in  the  managerial 
chain,  the  more  difficult  it  is  to  balance  all  three.  The  Nature  of  Measures  figure  used  by 
Simons  (Figure  4)  provides  a  template  to  follow  when  evaluating  objectivity, 
completeness,  and  responsiveness  of  perfonnance  metrics. 
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Figure  4.  Nature  of  Measures  (Simons,  2000,  p.  236) 


Lastly,  the  metric  has  to  be  linked  to  value.  The  information  contained  in  the 
metric  should  be  useful  for  the  managers  and  should  aid  in  decision  making.  The 
information  provided  by  the  metrics  should  enable  managers  to  be  more  effective  and  in 
the  end,  run  a  more  effective  organization.  In  essence,  the  metrics  should  provide 
valuable  information  useful  in  running  the  business  and  infonnative  in  getting  from  “A  to 
Z.” 

In  order  to  maximize  the  benefits  gained  from  measuring  performance,  managers 
must  take  care  in  selecting  and  evaluating  specific  perfonnance  metrics.  Specifically, 
performance  metrics  should  be  aligned  with  strategy,  be  objective,  complete  and 
responsive,  and  be  linked  to  value. 
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III.  METHODOLOGY 


The  author  conducted  a  series  of  interviews  with  the  Department  of  the  Navy 
Procurement  Metrics  points  of  contact  provided  by  DASN  ACQ.  DASN  ACQ  provided 
points  of  contact  for  the  ten  SYSCOMs  reporting  on  the  metrics.  Eventually  six-face-to- 
face  interviews  were  performed  over  the  course  of  three  days.  The  interviews  varied 
from  one-on-one,  one-on-two,  and  occasionally  involved  talking  to  three  or  more  people. 
The  additional  individuals  were  identified  by  the  points  of  contacts  as  having  valuable 
insights  about  the  metrics.  In  total,  the  author  interviewed  thirteen  individuals  face-to- 
face.  The  interviews  were  conducted  at  six  SYSCOM  sites  and  lasted,  on  average,  an 
hour  and  a  half.  There  were  no  telephone  interviews  or  follow-up  interviews. 

The  interviewees  had  varied  familiarity  with  the  performance  metrics  and  job 
experience.  This  ranged  from  approximately  2-3  weeks  as  the  DoN  Procurement  Metrics 
point  of  contact  to  being  involved  in  the  process  throughout  the  history  of  the  metrics 
effort.  Regardless  of  this  factor,  every  interviewee  offered  insight  into  the  metrics.  Each 
interviewee  had  his  or  her  own  specific  tone  about  the  metrics;  however  commonality  of 
issues  with  the  metrics  was  apparent  across  all  of  the  interviews. 

The  general  framework  used  to  evaluate  the  metrics  consists  of  three  interview 
questions:  (1)  Do  the  metrics  align  with  strategy?  (2)  Can  the  metrics  be  measured 
effectively?  and  (3)  Are  the  metrics  linked  to  value?  The  interviews  provided  key 
information  that  was  useful  in  answering  all  three  of  these  questions.  The  information 
also  provided  general  views  on  how  the  overall  metrics  process  could  be  improved  and 
specifically  how  the  metrics  could  become  more  valuable  for  the  HCAs. 

Table  1  is  a  summary  of  the  DoN  Procurement  Metrics  evaluated  in  this  project. 
The  following  section  summarizes  the  interviewees’  and  author’s  evaluation  of  the 
current  performance  metrics.  The  author’s  evaluation  of  the  metrics  represents  the 
synergy  of  perfonnance  measurement  theory  and  information  gained  during  the 
interviews. 
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FOCUS 

AREA 

TIER  1  METRIC 

TIER  2  METRIC 

TIER  3 
METRIC 

TIER  4 
METRIC 

DESCRIPTION  /  FORMULA 

FREQUENCY 

COLLECTED 

BY 

Customer 

Customer  Satisfaction  Index 

Average  Index  from  Customer  Satisfaction  Survey 

Annual 

HCA 

People 

Human  Capital  Index 

Employee  Survey 

Average  Index  from  Employee  Satifaction  Survey 

Annual 

HCA 

Workforce  Stability 

Percent  difference  between  authorized  end-strength 
and  actual  employees  onboard 

Quarterly 

HCA 

Qualifications 

Continous 

Learning 

Percent  employees  meet  their  CL  requirements  and 
hold  valid  certificates 

Annual 

DASN  ACQ 

DAWIA 

Percent  DAWIA  certified  to  or  above  level  required  by 
their  position 

Quarterly 

DASN  ACQ 

Acquisition 

Professional 

Community 

Percent  of  incumbents  of  CAPs  who  are  APC 
members 

Quarterly 

DASN  ACQ 

Process 

Process  Improvement  Index 

E2E  Metrics 

Percent  of  automated  procurement  transactions 
(defined  on  PEO-ARBS  website) 

Quarterly 

DASN  ACQ 

Efficiency 

Improvement 

P-Card 

Delinquency 

60  days  delinquent/total  transactions 

Quarterly 

DASN  ACQ 

DD1716 

Average  turnaround  time  from  receipt  to  disposition  of 
DD1716  (notice  of  contract  deficiencies) 

Quarterly 

DASN  ACQ 

Interest 

Penalties 

Ratio  YTD  interest  penalties/penalties  same  period 
previous  FY 

Quarterly 

DASN  ACQ 

Cycle  Time 

Average  cycle  time  as  the  term  is  defined  in  glossary 
for  ACAT  1  and  II  Program  contracts 

Annual 

HCA 

Consolidate  Services 
Contract 

Services  contracts  awarded  YTD/awarded  previous 

FY 

Quarterly 

DASN  ACQ 

Financial 

Contribution  Index 

Procurement 
Direct/Indirect  Ratio 

Procurement  direct  labor  dollars/total  procurement 
labor  dollars 

Annual 

HCA 

Industry  Spend 
Analysis 

Competition 

Percent  dollars  awarded  competitively/total  available 

Quarterly 

DASN  ACQ 

Small  Business 

Percent  dollars  awarded  to  small  businesses/total 
dollars  awarded  and  available  for  small  business 

Quarterly 

DASN  ACQ 

Commercial 

Items 

Actions 

Percent  contracts  awarded  for  commercial  items/total 
contracts  awarded 

Quarterly 

DASN  ACQ 

Dollars 

Percent  dollars  awarded  for  commercial  items/  dollars 
awarded  for  commercial  items  in  '99 

Quarterly 

DASN  ACQ 

Performance 
Based  Services 
Contracting 

Actions 

Percent  actions  awarded  for  performance  based 
services/total  actions  awarded  for  services 

Quarterly 

DASN  ACQ 

Dollars 

Percent  dollars  awarded  for  performance  based 
services/total  dollars  awarded  for  services 

Quarterly 

DASN  ACQ 

Value 

Product  Unit  Costing  (PUC) 

Large  Contracts 

Procurement  salary  dollars/contract  action 
dollars*(customer  service  survey  index)2,  for  contracts 
>  $100k 

Annual 

HCA 

Simplified  Acquisition 
Procedures 

Procurement  salary  dollars/contract  action 
dollars*(customer  service  survey  index)2,  for  contracts 
<  $100k 

Annual 

HCA 

Table  1.  DoN  Procurement  Metrics  Summary 
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IV.  PROCUREMENT  METRICS  ANALYSIS 


The  Procurement  Metrics  analysis  is  divided  into  four  sections.  The  first  three 
sections  answer  the  three  questions  described  as  the  general  framework  for  evaluating  the 
metrics:  1)  Do  the  metrics  align  with  strategy?  2)  Can  the  metrics  be  measured 
effectively?  and  3)  Are  the  metrics  linked  to  value?  The  fourth  section  of  this  chapter 
describes  other  findings  that  do  not  specifically  address  the  three  questions. 


A.  DO  THE  METRICS  ALIGN  WITH  STRATEGY? 


The  metrics  are  categorized  into  five  focus  areas;  Customer,  People,  Process, 
Financial,  and  Value.  Each  focus  area  has  an  associated  strategic  theme.  The  first 
question  evaluates  whether  the  metrics  link  back  to  the  overall  strategy  of  each  focus 
area.  If  they  are  not  aligned  with  strategy  then  there  is  a  fundamental  disconnect. 

1.  Customer 


Metric 


Customer  Satisfaction  Index 


The  Customer  focus  area  uses  only  one  metric,  the  Customer  Satisfaction  Index. 
The  strategic  theme  is  to  “Deliver  unprecedented  offensive  power,  defensive  assurance 
and  operational  independence  to  the  Joint  Force  Commanders”  (PMG,  vl4.3,  p.  7).  Put 
another  way,  the  strategy  is  to  “give  the  customers  what  they  want.”  The  index  is  an 
average  of  the  scores  from  customer  satisfaction  surveys  which  address  four  specific 
customer  satisfaction  themes  identified  in  the  Procurement  Metrics  Guide.  The  Customer 
Satisfaction  Index  metric  is  aligned  with  strategy;  the  SYSCOMs  want  to  know  how  well 
they  are  serving  their  customers.  However,  this  does  not  mean  that  the  metric  is  flawless. 
The  metric  is  still  subject  to  the  tests  of  whether  it  is  linked  to  value  and  can  be  accurately 
measured. 
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2.  People 


Metrics 

Employee  Survey 

Workforce  Stability 

Continuous  Learning 

DAWIA  Certification 

Acquisition  Professional  Community 

The  People  focus  area  has  five  associated  metrics.  The  five  metrics  are  the 
Employee  Survey,  Workforce  Stability,  Continuous  Learning,  DAWIA  Certification,  and 
Acquisition  Professional  Community.  The  strategic  theme  is  to  “Maintain  world-class 
workforce”  (PMG,  vl4.3,  p.  7).  All  of  these  metrics  align  with  strategy.  The  employee 
survey  is  intended  to  measure  the  employee’s  overall  satisfaction  with  the  organization 
which  is  an  important  component  of  maintaining  a  world-class  workforce.  A  satisfied 
workforce  is  likely  to  be  more  stable  and  keeping  the  workforce  stable  (providing  they 
are  well  trained)  is  certainly  a  part  of  maintaining  a  world-class  workforce.  The  last  three 
metrics  are  in  place  to  ensure  that  employees  have  the  knowledge  and  necessary  skills  to 
be  effective  in  their  positions,  making  Workforce  Stability  that  much  more  meaningful. 
Continuous  Learning  and  the  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 
metrics  measure  the  percentage  of  employees  who  have  their  required  certification. 
Continuous  Learning  certification  requires  job  relevant  “learning”  to  be  achieved  each 
year.  The  level  of  DAWIA  certification  needed  depends  on  the  position  that  the  employee 
holds.  In  order  to  be  certified,  employees  need  to  be  certified  to  their  level  position  or 
higher.  The  Acquisition  Professional  Community  metric,  collected  quarterly,  identifies 
the  percentage  of  incumbents  in  Critical  Acquisition  Positions  who  are  members  of  the 
Acquisition  Professional  Community  (APC)  compared  to  the  total  number  of  incumbents 
in  Critical  Acquisition  Positions.  The  PMG  states,  “Selection  to  the  APC  is  based  on 
meeting  specific  training,  education,  experience  and  grade  requirements.”  These 
requirements  are  controlled  by  the  Director  of  Acquisition  Career  Management  (DACM). 
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3.  Process 


Metrics 

E2E  Metric 

P-Card  Delinquency 

DD1716 

Interest  Penalties 

Cycle  Time 

Consolidate  Service  Contract 

The  third  focus  area  is  Process.  There  are  six  associated  metrics  in  this  focus 
area.  The  strategic  theme  for  Process  is  to  “Continuously  improve  efficiency”  (PMG, 
vl4.3,  p.  7).  The  metrics  associated  with  the  Process  focus  area  are  E2E  Metrics,  P-Card 
Delinquency,  DD1716,  Interest  Penalties,  Cycle  Time,  and  Consolidate  Service  Contract. 

Business  transactions  are  moving  in  the  direction  of  paperless  transactions.  The 
Navy  has  set  up  a  website  (www.peoarbs.navy.mil)  which  it  refers  to  as  the  Paperless 
Acquisition  Office  of  the  Navy.  The  goal  of  the  website  is  “to  simplify  and  modernize 
the  Navy  acquisition  process  in  the  area  of  contract  writing,  administration,  finance  and 
auditing.”  The  E2E  Metric  measures  the  percentage  of  total  procurement  transactions 
that  are  automated,  or  the  ratio  of  automated  procurement  transactions  over  total  number 
of  procurement  transactions.  Automation  in  this  case  improves  efficiency  because  it 
reduces  time  spent  filling  out  and  sending  paper  procurement  transactions.  This  metric 
links  to  the  strategy  of  the  Process  focus  area. 

The  DoD  Purchase  Card  Program  Management  Office  sent  out  a  memorandum  on 
17  SEP  1999  regarding  the  issue  of  Purchase  Card  Delinquency  policies  and  the  need  to 
keep  delinquent  payments  (over  60  days)  to  a  minimum.  The  P-Card  Delinquency  metric 
is  the  percentage  of  dollars  of  delinquent  P-card  payments  compared  to  the  total  purchase 
card  dollars.  The  metric  is  linked  with  the  overall  strategy  for  the  Process  focus  area 
because  decreasing  delinquent  P-Card  payments  results  in  an  increased  efficiency  of 
payments.  This  metric  is  consistent  with  DoD  policy  as  well. 

The  DD1716  Metric  measures  the  average  turnaround  time  of  the  DD1716  form, 
which  is  a  contract  deficiency  report.  The  quick  resolution  of  the  deficiency  can  limit  the 
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amount  of  interest  charged.  The  sooner  the  turnaround  time  on  these  forms  the  less 
money  that  has  to  be  paid  out  and  more  efficient  the  process  which  directly  links  to  the 
strategy. 

Interest  penalties  cost  the  Navy  and  the  DoD  as  a  whole  millions  of  dollar  every 
year.  Paying  interest  is  essentially  wasted  money  that  results  from  inefficient  practices 
and/or  contract  mistakes.  Not  only  does  the  Interest  Penalty  metric  provide  the 
opportunity  to  monitor  dollar  savings,  but  it  is  also  a  means  to  monitor  if  the  process  is 
becoming  more  efficient.  Increasing  dollar  savings  and  efficiency  work  hand  in  hand 
towards  the  ultimate  objective:  an  efficient  environment  which  becomes  an  effective  cost 
savings  tool.  This  metric  is  collected  quarterly  and  compares  the  current  year’s  interest 
penalties  paid  to  date  to  the  same  period  of  the  former  fiscal  year.  The  output  is  a  ratio 
which  hopefully  falls  below  100  percent.  This  metric,  as  well  as  Consolidate  Service 
Contract  and  the  Industry  Spend  Analysis — Commercial  Items  (actions  &  dollars)  metrics 
compare  current  year  to  last  year,  so  the  base  needs  to  be  adjusted  every  year. 

Cycle  time  is  very  important  in  the  procurement  world  and  there  are  many  factors 
that  influence  it  (e.g.,  technological  maturity  of  the  proposed  system).  Reducing  cycle 
time  can  save  money.  More  importantly,  a  reduction  in  cycle  time  gets  the  products  or 
services  to  the  war  fighter  sooner.  A  reduction  in  cycle  time  can  give  the  United  States 
military  a  distinct  advantage  over  other  militaries.  The  Cycle  Time  metric  is  defined  as 
“total  time  in  days  between  the  acquisition  start  date  and  end  date  (only  applicable  to 
AC  AT  I  and  II  acquisitions)”  (PMG,  vl4.3,  p.  17).  Definitions  of  the  start  and  end  date 
are  available  in  the  Procurement  Metrics  Guide.  The  Cycle  Time  metric  is  linked  to  the 
Process  focus  area  strategy. 

The  Consolidate  Service  Contract  metric  is  the  last  one  under  the  Process  focus 
area.  This  metric  is  collected  quarterly  and  is  measured  in  a  manner  that  is  similar  to  the 
Interest  Penalty  metric.  It  is  a  ratio  of  the  number  of  services  contracts  awarded  year-to- 
date  as  compared  to  the  same  time  the  previous  year.  The  goal  for  the  ratio  is  for  it  to  be 
less  than  100  percent,  indicating  that  some  of  the  services  contracts  have  been 
consolidated  or  eliminated.  This  metric  is  aligned  with  the  overall  strategy  of  the  Process 
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focus  area  because  fewer  contracts  will  have  to  be  maintained  and  administered  and  this 
increases  the  potential  for  more  efficient  operations. 


4.  Financial 


Metrics 

Procurement  Direct/Indirect  Ratio 

Industry  Spend  Analysis — Competition 

Industry  Spend  Analysis — Small  Business 

Industry  Spend  Analysis — Commercial  Items — Actions 

Industry  Spend  Analysis — Commercial  Items — Dollars 

Industry  Spend  Analysis — Performance  Based  Services — Actions 

Industry  Spend  Analysis — Perfonnance  Based  Services — Dollars 

The  Financial  strategic  theme  is  to  “Maximize  return  on  taxpayer  investment” 
(PMG,  vl4.3,  p.  7).  The  financial  focus  area  has  seven  associated  metrics.  The  metrics 
are  (1)  Procurement  Direct/Indirect  Ratio,  (2)  Industry  Spend  Analysis — Competition,  (3) 
Industry  Spend  Analysis — Small  Business,  (4)  Industry  Spend  Analysis — Commercial 
Items-Actions,  (5)  Industry  Spend  Analysis — Commercial  Items-DoIIars,  (6)  Industry 
Spend  Analysis — Performance  Based  Sendees  Acquisition  Contracting- Actions,  and  (7) 
Industry  Spend  Analysis — Performance  Based  Services  Acquisition  Contracting-Dollars) . 

The  Procurement  Direct/Indirect  Ratio  metric  is  an  attempt  at  measuring  the 
“bang  for  the  buck”  of  each  SYSCOM  when  it  comes  to  how  much  support  is  needed  to 
perform  the  procurement  function.  The  metric  is  a  ratio  of  the  total  number  of  direct 
labor  employees  as  compared  to  the  total  number  of  employees  in  the  procurement 
organization.  Direct  labor  employees  are  defined  as  “employees  whose  primary 
responsibilities  are  directly  involved  with  efforts  required  to  perform  the  procurement 
function”  (PMG,  vl4.3,  p.  17).  The  idea  is  to  provide  the  necessary  support  to  efficiently 
and  effectively  accomplish  all  of  the  necessary  contracting  activities  while  keeping  the 
number  of  supporting  positions  under  control.  The  strategy  of  maximizing  the  return  on 
taxpayer  investment  follows  logically,  but  issues  with  this  metric  are  discussed  in  later 
sections. 

The  Industry  Spend  Analysis — Competition  metric  has  a  direct  link  to  the  strategy 
of  the  Financial  focus  area,  maximize  return  on  taxpayer  investment.  This  metric  is  the 
ratio  of  total  procurement  dollars  awarded  competitively  as  compared  to  the  total 
procurement  dollars  available  for  competition.  The  benefits  of  competition;  cost 
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reduction,  improved  quality,  and  innovation,  are  widely  known.  There  are  instances  of 
course  when  competition  is  not  the  most  advisable  way  to  proceed  (e.g.,  when  only  one 
source  is  able  to  provide  the  product  or  service,  when  time  is  a  critical  factor,  or  when  a 
dependable  long-term  relationship  is  necessary). 

The  Industry  Spend  Analysis — Small  Business  metric  links  to  the  strategy  of  the 
Federal  Acquisition  Regulations  regarding  small  business  opportunities  for  government 
contracts.  This  metric  does  not  clearly  link  to  the  financial  focus  area  strategy  of 
maximizing  taxpayer  investment.  However,  it  does  reflect  social  responsibility  and  the 
entrepreneurial  spirit  captured  in  the  Small  Business  Act.  The  spirit  of  the  Small 
Business  Act  is  to  encourage  small  business  participation  in  government  contracts  and 
specifically,  “small  business  concerns  shall  be  afforded  an  equitable  opportunity  to 
compete  for  all  contracts  that  they  can  perform  to  the  extent  consistent  with  the 
Government’s  interest”  (Federal  Acquisition  Regulation  [FAR],  19.202-1). 

The  Federal  Acquisition  Regulation  also  states: 

It  is  the  policy  of  the  Government  to  provide  maximum  practicable 
opportunities  in  its  acquisitions  to  small  business,  veteran-owned  small 
business,  service-disabled  veteran-owned  small  business,  HUBZone  small 
business,  small  disadvantaged  business,  and  women-owned  small  business 
concerns.  The  Small  Business  Administration  (SBA)  counsels  and  assists 
small  business  concerns  and  assists  contracting  personnel  to  ensure  that  a 
fair  proportion  of  contracts  for  supplies  and  services  is  placed  with  small 
business  (FAR,  19.201). 

The  Industry  Spend  Analysis — Commercial  Items  metrics  (Actions  &  Dollars) 
both  are  linked  to  the  Financial  focus  area  strategy  as  well  as  that  of  the  Federal 
Acquisition  Regulations.  Some  of  the  benefits  of  using  Commercial  and  Non- 
Developmental  items  (CANDI)  is  the  speed  at  which  these  items  can  be  procured,  cost 
savings,  and  proven  technology  which  mitigates  risk.  Using  CANDI  for  technologically 
driven  products  maximizes  taxpayer  investment  because  often  times  commercial  demand 
for  these  items  may  quicken  the  evolution  and  fielding  beyond  the  pace  at  which 
government  research  and  development  could  produce  the  same  items.  The  Actions  metric 
is  the  “ratio  of  the  number  of  contracts  awarded  year  to  date  for  commercial  items  as 
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compared  to  the  total  number  of  contracts  awarded”  (PMG  vl4.3,  p.  13).  The  Dollars 
metric  is  the  “ratio  of  HCA  total  product  dollars  awarded  year  to  date  for  commercial 
items  as  compared  to  the  total  product  dollars  awarded  in  1999,  during  the  comparable 
period,  for  commercial  items”  (PMG,  vl4.3,  p.  13). 

The  Federal  Acquisition  Regulation  states: 

Agencies  shall — (a)  Conduct  market  research  to  determine  whether 
commercial  items  or  nondevelopmental  items  are  available  that  could 
meet  the  agency’s  requirements;  (b)  Acquire  commercial  items  or 
nondevelopmental  items  when  they  are  available  to  meet  the  needs  of  the 
agency;  and  (c)  Require  prime  contractors  and  subcontractors  at  all  tiers  to 
incorporate,  to  the  maximum  extent  practicable,  commercial  items  or 
nondevelopmental  items  as  components  of  items  supplied  to  the  agency 
(FAR,  12.101). 

The  Industry  Spend  Analysis — Performance  Based  Services  Acquisition  (PBSA) 
Contracting  metrics  (Actions  and  Dollars)  are  aligned  with  focus  area  strategy  as  well  as 
DoD  policy.  The  motivation  for  these  metrics  is  that  people  working  within  contracting 
do  not  need  to  get  bogged  down  in  the  details  of  writing  specific  processes  of  how  a 
service  should  be  perfonned.  The  contractor  is  supposed  to  be  the  expert,  so  give  the 
contractor  specifics  on  what  the  end  result  of  the  service  should  be,  but  not  the  minutia  of 
how  to  do  it.  The  effects  of  PBSA  should  be  cost  savings  and  a  more  effective  and 
efficient  use  of  limited  DoD  resources  because  the  focus  shifts  to  execution. 

PBSA  involves  acquisition  strategies,  methods,  and  techniques  that 
describe  and  communicate  measurable  outcomes  rather  than  direct 
performance  processes.  It  is  structured  around  defining  a  service 
requirement  in  terms  of  performance  objectives  and  providing  contractors 
the  latitude  to  determine  how  to  meet  those  objectives.  Simply  put,  it  is  a 
method  for  acquiring  what  is  required  and  placing  the  responsibility  for 
how  it  is  accomplished  on  the  contractor  (Guidebook  for  PBSA  in  the 
Department  of  Defense,  p.  6). 

The  PBSA  Actions  metric  is  defined  as  the  ratio  of  HCA  total  PBSA  awarded 
actions  as  compared  to  total  awarded  actions  for  services.  The  PBSA  Dollars  metric  is 
defined  as  the  ratio  of  HCA  total  dollars  awarded  for  PBSA  contracts  as  compared  to 
HCA  total  dollars  awarded  for  all  services  contracts. 
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5. 


Value 


Metrics 

Performance  Unit  Costing — Large  Contracts 

Performance  Unit  Costing — Simplified  Acquisition  Procedures 

The  fifth  and  final  focus  area  is  Value.  The  strategic  theme  of  the  Value  focus 
area  is  the  same  as  the  Financial  focus  area:  Maximize  return  on  taxpayer  investment. 

The  objective  for  this  focus  area  is  to  “Balance  quality,  cost  and  productivity”  (PMG, 
vl4.3,  p.  8).  There  are  two  metrics  in  this  focus  area  and  both  support  the  strategic 
theme.  The  metrics  are  Performance  Unit  Costing  (PUC) — Large  Contracts  and  PUC — 
Simplified  Acquisition  Procedures.  Both  metrics  use  the  same  formula:  HCA  total 
procurement  salary  dollars  (C),  divided  by  the  product  of  contract  action  dollars  (O)  and 
the  square  of  the  customer  satisfaction  index  (I  ).  The  resultant  formula  is:  [C/(0*I  )]. 
The  two  metrics  have  different  definitions  for  “O,”  the  contract  action  dollars,  because  of 
the  threshold  between  Large  Contracts  and  Simplified  Acquisition  Procedures.  The 
Simplified  Acquisition  Threshold  is  defined  by  the  FAR  as  follows: 

Simplified  acquisition  threshold  means  $100,000,  except  for  acquisitions 
of  supplies  or  services  that,  as  determined  by  the  head  of  the  agency,  are  to 
be  used  to  support  a  contingency  operation  or  to  facilitate  defense  against 
or  recovery  from  nuclear,  biological,  chemical,  or  radiological  attack 
(FAR,  2. 101). 

The  PUC-Large  Contracts  metric  is  measured  by  taking  “HCA  total  procurement 
salary  dollars  (C)  divided  by  contract  action  dollars  (O)  for  contracts  valued  at  greater 
than  $100k  multiplied  by  the  square  of  the  customer  satisfaction  index  (I):  [C/(0*I2)]” 
(PMG  vl4.3,  14).  The  PUC-Simplified  Acquisition  Procedures  metric  is  calculated  the 
same  way,  just  using  the  “$  100k  and  less”  threshold  to  determine  the  “O”  variable.  The 
motivation  for  these  metrics  is  to  mesh  the  “bang  for  the  buck”  (cost  and  productivity) 
with  the  perceived  quality  by  the  customers.  The  output  is  a  number  (e.g.  .004959716), 
which  by  itself,  is  of  no  use.  It  can  be  of  some  use  if  several  other  factors  are  taken  into 
account  such  as  the  validity  of  the  Customer  Satisfaction  Index,  the  ability  to  accurately 
collect  total  procurement  salary  dollars  and  having  several  data  points  (a  trend)  to 
analyze. 
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B. 


CAN  THE  METRICS  BE  MEASURED  EFFECTIVELY? 


All  of  the  DoN  metrics  appear  to  be  aligned  with  the  strategies  of  the  focus  areas. 
One  of  the  strategies  is  to  give  the  war  fighters  what  they  need  when  they  need  it  and  a 
survey  could  possibly  capture  all  that  is  needed  to  ascertain  that,  but  it  could  also  fall 
short.  In  this  section,  the  metrics  will  be  evaluated  on  their  objectivity  and  completeness 
as  described  in  the  literature  review.  A  measure  is  objective  when  it  can  be 
independently  verified  and  is  complete  when  it  captures  all  aspects  of  construct. 
Determining  the  responsiveness  of  the  metrics  is  beyond  what  can  be  accomplished  in 
this  study.  Capturing  the  responsiveness  dimension  will  require  a  more  in  depth 
understanding  of  the  HCA  organization  structure  as  well  the  intricacies  and 
responsibilities  of  the  different  positions  within  that  organization.  To  reiterate  from  the 
literature  review,  a  responsive  measure  reflects  actions  that  a  manager  can  directly 
influence.  Most  of  these  metrics  encompass  actions  resulting  from  the  work  of  an  entire 
enterprise  and  therefore  it  is  difficult  to  ascertain  at  what  level  they  are  responsive  or  not 
responsive. 

1.  Customer 


Metric 


Customer  Satisfaction  Index 


Can  Customer  Satisfaction  be  measured  effectively?  It  depends  on  several  things. 
For  instance:  Who  is  filling  out  the  survey?  Is  it  the  same  person  as  last  year?  How  does 
this  person  typically  fill  out  a  survey:  accurately,  harshly,  or  lenient?  Is  this  the  same 
survey  that  was  given  out  last  year?  If  not,  how  similar  is  it  to  last  years,  does  it  cover 
the  same  general  customer  satisfaction  themes?  Does  a  survey  even  go  out  every  year  or 
was  last  year’s  survey  responses  used  for  this  year’s  metric?  What  was  your  response 
rate?  All  of  these  questions  offer  a  look  into  whether  or  not  the  Customer  Satisfaction 
Index  metric  can  be  an  effective  measure  as  well  as  a  tool  to  analyze  trends  from  the 
metric. 


All  of  the  aforementioned  questions  about  the  surveys  identify  possible  risks  in 
using  the  surveys  in  trend  analysis.  Is  a  survey  better  than  none?  Absolutely.  Any  one 
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of  the  factors  identified  in  the  questions  by  itself  is  not  likely  to  affect  drastically  the 
validity  of  the  Customer  Satisfaction  Index,  but  put  together,  they  can  throw  off  the 
baseline  and  trend  analysis. 

The  questions  mainly  address  the  issue  of  objectiveness.  Surveys  are  inherently 
subjective,  but  if  they  are  performed  in  good  faith,  address  the  four  customer  satisfaction 
themes,  are  reported  on  accurately,  and  have  a  high  response  rates,  then  they  should  be  an 
effective  measure  of  customer  satisfaction. 

Is  the  Customer  Satisfaction  Index  complete;  does  it  capture  all  the  relevant 
attributes  of  achievement?  If  the  survey  covers  fully  the  four  customer  satisfaction 
themes  required,  as  well  as  goes  to  and  is  received  back  from  a  large  number  of 
customers  (high  response  rate),  as  described  above,  then  the  survey  should  capture  all  the 
attributes  of  Customer  Satisfaction.  The  interviewees  stated  that  customer  response  rate 
was  not  a  problem.  However,  there  is  currently  no  required  response  rate  delineated  in 
the  PMG  setting  the  bar  for  acceptable  survey  results. 

2.  People 


Metrics 

Employee  Survey 

Workforce  Stability 

Continuous  Learning 

DAWIA  Certification 

Acquisition  Professional  Community 

All  of  the  questions  that  can  be  asked  about  the  Customer  Satisfaction  Survey  can 
be  asked  about  the  Employee  Survey  and  they  lead  in  the  same  direction.  The 
characteristics  of  an  objective  and  complete  employee  survey  can  be  gleaned  from  the 
above  discussion  of  the  customer  satisfaction  survey.  That  is,  surveys  are  inherently 
subjective,  but  that  characteristic  can  be  minimized  through  the  appropriate  methods;  for 
instance,  requiring  a  high  response  rate  and  accurate  reporting.  No  specific  percentage 
for  response  rate  was  discussed  in  the  interviews,  but  one  interviewee  stated,  “Need  to 
put  in  place  a  certain  percentage  that  has  to  respond  for  it  to  be  valid.  The  survey  is 
anonymous,  so  it  is  hard  to  control.”  Also,  the  survey  is  complete  as  long  as  it  measures 
all  necessary  attributes  of  employee  satisfaction.  One  interviewee  mentioned  that  their 
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employee  survey  is  sent  to  the  entire  SYSCOM,  not  just  the  HCA,  so  it  is  not  just 
measuring  the  HCA’s  employee  satisfaction. 

Workforce  Stability  is  an  objective  measure  unless  estimates  are  used;  however, 
this  metric  is  not  complete  as  it  currently  stands.  The  metric  does  not  capture  significant 
elements  of  workforce  stability;  rather  it  captures  just  a  limited  scope.  It  should  be 
possible  to  effectively  measure  Workforce  Stability,  but  that  is  not  what  is  being 
measured  by  using  this  metric.  This  metric  measures  the  percent  difference  between  the 
authorized  end-strength  and  the  actual  number  of  employees  onboard  as  of  the  end  date 
of  the  quarter.  This  measures  how  “full”  the  organization  is,  not  how  stable  the 
workforce  is.  Even  with  several  data  points  of  this  metric  creating  a  trend,  it  still  is  not 
very  useful  unless  you  are  assessing  how  full  an  organization  is  over  time.  The  measure 
tangentially  relates  to  stability.  If  during  a  period  of  time  the  percent  difference  stayed 
relatively  stable,  it  could  be  that  there  was  stability  in  the  workforce.  It  could  also  mean 
that  each  time  you  measured  stability  using  this  formula,  the  number  of  employees  and 
end-strength  remained  about  the  same,  but  that  there  was  a  100  percent  turnover  of  the 
workforce.  This  is  not  a  stable  workforce,  it  has  high  turnover  which  is  almost  certainly 
detrimental  to  the  organization’s  effectiveness. 

The  next  three  metrics  under  the  People  Focus  Area,  Qualifications:  (1) 
Continuous  Learning,  (2)  DAWIA  certification,  and  (3)  Acquisition  Professional 
Community  can  be  measured  effectively.  The  data  for  these  metrics  are  pulled  by  DASN 
ACQ  from  the  Data  Acquisition  and  Control  Manager  Mission  Information  System 
database  (DACM  MIS  database).  As  long  as  the  infonnation  is  correctly  inputted  in  the 
database  the  resulting  metric  should  be  accurate  and  therefore  make  the  metric  an 
objective  one.  These  metrics  also  present  a  full  picture  of  the  level  of  qualifications  that 
the  procurement  workforce  making  the  measures  relatively  complete. 
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3.  Process 


Metrics 

E2E  Metric 

P-Card  Delinquency 

DD1716 

Interest  Penalties 

Cycle  Time 

Consolidate  Service  Contract 

The  E2E  metric  can  be  effectively  measured.  This  metric  is  objective  because 
information  for  the  metric  is  taken  from  the  same  database  for  each  SYSCOM  and 
therefore,  can  be  independently  verified.  As  is  the  case  with  most  of  the  metrics  that  are 
being  collected  from  databases,  the  accuracy  of  the  infonnation  entered  into  the  database 
is  paramount  to  getting  an  accurate  reflection  of  the  SYSCOM.  The  P-Card  Delinquency 
metric  is  affected  by  the  same  circumstances  as  the  E2E  metric.  It  is  effectively 
measured  through  the  Navy’s  e-Business  Office  and  is  objective.  The  Consolidate 
Sendee  Contract  metric  is  collected  by  DASN  ACQ  from  the  Procurement  Management 
Reporting  System  (PMRS).  All  three  metrics;  E2E ,  P-Card  Delinquency,  and 
Consolidate  Service  Contract,  are  narrowly  focused  and  inclusive,  making  the 
measurement  of  the  metric  complete. 

The  Efficiency  Improvement  metric  of  DD1716  is  collected  by  DASN  ACQ  from 
the  Navy-Air  Force  Interface  (NAFI)  website.  This  metric  is  relatively  objective  because 
the  data  is  pulled  from  the  NAFI  website,  but  there  is  a  tinge  of  subjectivity  surrounding 
what  the  author  views  as  the  potential  for  gaming  the  system,  which  is  discussed  in  a  later 
section.  This  metric,  if  the  receiving  date  entered  into  the  NAFI  website  is  agreed  upon, 
is  complete  in  its  measure  of  the  efficiency  of  DD 17 16  resolution. 

The  issues  of  objectivity  and  completeness  for  the  Cycle  Time  metric  are  similar 
to  the  DD1 716  metric.  This  metric  is  reported  by  the  HCAs  instead  of  being  collected 
from  a  database,  increasing  its  subjectivity.  Since  there  are  several  determinates  when 
deciding  the  actual  start  and  end  date  of  the  metric,  the  chances  of  gaming  increase.  It  is 
a  relatively  complete  measure  of  the  start  to  the  end  of  acquisition  time. 
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4.  Financial 


Metrics 

Procurement  Direct/Indirect  Ratio 

Industry  Spend  Analysis — Competition 

Industry  Spend  Analysis — Small  Business 

Industry  Spend  Analysis — Commercial  Items — Actions 

Industry  Spend  Analysis — Commercial  Items — Dollars 

Industry  Spend  Analysis — Perfonnance  Based  Services — Actions 

Industry  Spend  Analysis — Perfonnance  Based  Services — Dollars 

The  Procurement  Direct/Indirect  Ratio  metric  is  a  measure  that  has  potential.  As 
stated  above  in  the  “Aligning  with  Strategy”  section,  this  metric  is  designed  to  measure 
how  much  support  is  needed  to  carry  out  the  procurement  function  for  each  SYSCOM. 
The  metric  is  likely  to  differ  between  the  SYSCOMs.  That  is  appropriate.  One 
SYSCOM  could  mainly  be  making  small  purchases  and  the  other  purchasing  Aircraft 
Carriers.  The  two  SYSCOMs  in  this  example  would  have  different  ratios  given  a  need 
for  different  levels  of  various  activities  and  processes.  Even  if  the  ratios  differ  across 
SYSCOMs,  a  trend  analysis  within  the  SYSCOMs  is  beneficial. 

Given  how  this  metric  is  collected,  it  is  not  totally  objective.  There  needs  to  be 
very  specific  instruction  on  which  positions  should  be  reflected  in  this  metric.  Precision 
and  accuracy  in  collecting  the  data  for  this  metric  will  ensure  it  is  objective.  One 
interviewee  stated,  “Direct/Indirect  [is]  a  little  bit  subjective.  The  infonnation  is  not  easy 
to  get.”  Another  interviewee  stated,  “Does  anyone  track  procurement  salary? 

Information  is  there,  but  not  broken  down  into  sections  like  procurement  and  further  into 
direct  and  nondirect.”  Reflecting  the  need  for  clarification  on  definitions  in  regards  to 
this  metric,  one  interviewee  stated,  “There  is  confusing  language  in  the  definition  of 
tenns,  people  need  to  clearly  understand  what  is  being  asked  for.”  This  metric  gives  a 
relatively  complete  picture  of  how  much  indirect  support  is  needed  by  SYSCOM  in  order 
to  perfonn  their  level  of  procurement. 

The  Industry  Spend  Analysis — Competition  metric  can  be  effectively  measured. 
All  of  the  information  needed  to  calculate  the  metric  is  in  the  Procurement  Management 
Reporting  System  (PMRS)  and  is  pulled  by  DASN  ACQ  which  makes  it  an  objective 
measure.  The  same  also  holds  true  for  the  following  five  Industry  Spend  Analysis 
metrics:  (1)  Small  Business,  (2)  Commercial  Items — Actions,  (3)  Commercial  Items — 
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Dollars,  (4)  Performance  Based  Sendees  Acquisition  Contracting — Actions  and  (5) 
Performance  Based  Services  Acquisition  Contracting — Dollars.  These  metrics  are  all 
collected  from  PMRS  by  DASN  ACQ.  All  of  these  metrics  are  complete;  they  capture  all 
the  relevant  information  that  is  needed  to  get  a  useful  view  of  how  the  Navy’s 
procurement  function  is  performing  in  each  of  these  areas. 


5.  Value 


Metrics 

Performance  Unit  Costing — Large  Contracts 

Performance  Unit  Costing — Simplified  Acquisition  Procedures 

The  Performance  Unit  Costing  (PUC)  metrics,  Large  Contracts  and  Simplified 
Acquisition  Procedures,  can  be  measured,  but  the  degree  of  the  measure’s  precision  is  in 
question.  First,  the  Customer  Satisfaction  Index  is  used  in  the  formula.  This  is  not  a 
completely  objective  variable,  so  it  adds  some  subjectivity  into  these  metrics.  This  index 
is  squared  in  the  formula  to  increase  its  affect  on  the  output  of  the  equation. 

Second,  because  the  data  available  for  procurement  salary  dollars  to  the 
SYSCOMs  differs,  the  use  of  reasonable  estimation  is  an  option  given  by  the  PMG.  This 
makes  the  likelihood  of  independent  verification  questionable.  One  interviewee  stated, 
“Procurement  salaries  can  be  subjective,  [you]  need  an  unbiased  source  that  they  can  pull 
from.” 

The  contract  action  dollars,  variable  “O,”  is  defined  as  “The  total  net  value  of 
obligations  under  contract  actions”  (PMG,  vl4.3,  p.  16).  This  is  a  relatively  objective 
measure  with  no  estimation  needed. 

The  PUC  metrics  are  intended  to  be  a  complete  measure  of  value  and  specifically 
balance  quality,  cost  and  productivity.  The  metrics,  although  somewhat  abstract  and  not 
very  meaningful  without  trend  analysis,  do  blend  all  three  of  these  elements  together 
using  the  fonnula.  The  HCA  representatives  generally  understood  the  concept  and  why 
the  formula  was  created  the  way  it  was,  but  were  concerned  with  the  meaning  of  the 
output. 
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C.  ARE  THE  METRICS  LINKED  TO  VALUE? 

All  of  the  HCAs  interviewed  thought  that  feedback  from  DASN  ACQ  was 
significantly  lacking  on  (1)  the  DoN  Procurement  Metrics  in  general  and  (2)  specific 
feedback  on  the  metrics  DASN  ACQ  collects  on  each  of  the  SYSCOMs.  In  the  HCAs 
view,  if  they  are  not  collecting  the  data  for  a  metric,  then  it  does  not  even  exist  because 
they  have  not  seen  any  feedback  on  the  fourteen  metrics  that  DASN  ACQ  collects. 
Therefore  those  fourteen  metrics  are  of  little  value  to  the  HCAs,  regardless  if  they  have 
value  for  DASN  ACQ. 

Being  the  POCs  for  the  DoN  Procurement  Metrics  and  collecting  the  metrics  is  a 
collateral  duty  for  all  of  the  POCs.  They  receive  no  additional  resources  to  aid  in  the  data 
calls.  The  collection  of  these  metrics  is  a  time-consuming  process.  Collecting  the  data 
for  the  metrics  expends  valuable  resources  and  it  is  not  clear  to  the  HCAs  if  the  “cost”  is 
worth  the  benefit  or  value  the  metrics  provide  to  the  SYSCOMs  and  DASN  ACQ. 

An  issue  brought  up  in  one  of  the  interviews  was  whether  or  not  DASN  ACQ  had 
any  idea  of  how  much  it  cost  to  develop,  monitor  and  collect  the  data  on  the  metrics.  So, 
what  is  the  benefit  [value]  of  the  metrics  and  is  this  value  gained  from  the  metrics  worth 
the  resources  expended  in  managing,  monitoring,  and  collecting  them?  The  answer  to 
this  question  was  not  completely  discernible  from  any  of  the  interviews  conducted  with 
the  HCA  representatives  because  for  the  most  part  they  were  asking  of  what  value  are  the 
metrics  to  DASN  ACQ. 

The  majority  of  the  metrics  are  collected  on  an  annual  basis.  Of  the  seven  metrics 
that  the  HCAs  report  on,  only  one  is  reported  quarterly,  Workforce  Stability.  Reporting 
on  the  specific  metrics  annually  makes  sense  because  of  the  nature  of  procurements  due 
to  yearly  appropriations  from  Congress  and  the  fiscal  year  cycle.  However,  there  are  two 
mind-sets  at  odds  with  each  other.  The  first  is  that  sometimes  employees  are  not  in  their 
positions  long  enough  for  annual  metrics  to  be  of  any  value  in  a  managerial  sense 
because  managers  might  only  see  one  or  two  sets  of  annual  data.  Seeing  limited  data 
does  not  provide  managers  with  much  value  and  does  not  give  them  the  ability  to  make 
decisions  and  manage  based  on  the  variances  observed.  The  counter  to  this  argument  is 
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that  if  there  was  past  data,  then  that  could  be  looked  at  in  conjunction  with  current  data. 
The  second  mind-set  is,  as  stated  above,  collecting  the  data  is  not  a  minor  task  and 
answering  data  calls  more  than  once  a  year  for  specific  metrics  would  be  extremely  time- 
consuming  if  not  impossible  without  additional  resources  in  place. 

The  PUC  metrics  {Large  Contracts  &  Simplified  Acquisition  Procedures )  were 
discussed  more  frequently  during  the  interview  process  than  other  metrics.  The  general 
concept  of  the  metric  was  understood,  but  the  HCA  representatives  did  not  see  much 
value  in  the  output.  An  interviewee  stated,  “Concept  of  PUC  is  nice,  but  real  world 
application  is  questionable.  Goes  back  to  how  it  is  used.  [A  result  of]  .003  down,  why 
did  that  happen?”  The  HCAs  need  more  feedback  on  the  two  PUC  metrics  and  what  the 
results  mean.  The  use  of  the  Customer  Satisfaction  Index  was  a  point  of  contention  with 
one  SYSCOM.  There  were  two  parts  to  this  point.  One,  this  metric  is  already  used 
independently  of  the  PUC  metrics.  Second,  the  weighting  of  the  Customer  Satisfaction 
Index  is  increased  by  squaring  it  which  increases  the  “role”  of  this  subjective  and 
incomplete  metric. 

Another  issue  regarding  these  two  metrics  {PUC-Large  Contracts  &  PUC- 
Simplified  Acquisition  Procedures)  involved  the  use  of  contract  action  dollars  in  the 
formula.  Two  of  the  HCA  representatives  from  the  SYSCOMs  would  prefer  to  use 
“contract  actions”  instead  of  “contract  action  dollars”  as  one  of  the  variables  in  the 
metric.  One  other  SYSCOM  was  strongly  against  “contract  actions”  replacing  “contract 
action  dollars.”  This  SYSCOM  felt  this  change  would  not  create  a  more  accurate 
reflection  of  their  performance  in  this  area,  stating,  “Clearly  do  not  use  actions.”  Another 
SYSCOM  felt  that  there  was  not  a  clear  answer,  stating,  “When  you  use  dollars  you 
could  be  seen  as  more  efficient  without  being.  [There  is]  not  much  control  over  number 
of  actions  because  a  lot  of  control  is  set  by  Congress’  appropriations.” 

Indication  of  the  metric’s  link  to  value  was  the  SYSCOMs  view  that  the  Navy 
Procurement  metrics  are  not  the  specific  ones  that  each  individual  SYSCOM  would  have 
come  up  with  to  tell  the  story  of  the  health  of  their  specific  organization,  but  that  was  not 
the  purpose  of  these  metrics.  One  SYSCOM  stated,  “We  wouldn’t  have  come  up  with  all 

of  these  metrics  if  you  asked,  ‘how  [our  SYSCOM]  is  doing?”’ 
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D.  ADDITIONAL  ANALYSIS 


It  became  clear  to  the  author  through  the  evaluation  of  the  metrics  that  there  were 
two  different  types  of  analysis,  design  and  use.  The  evaluation  of  the  metrics’  alignment 
with  strategy,  effective  measurement  (objectivity  and  completeness),  and  link  to  value,  all 
fall  under  the  “design”  of  this  management  control  system.  The  analysis  of  design 
captures  the  fundamental  structure  of  the  system  and  answers  the  questions:  Is  it  valid?  Is 
the  design  sufficient  enough  for  the  system  to  work?  This  section  discusses  findings 
related  to  the  “use”  of  the  metrics.  This  includes,  but  is  not  limited  to,  issues  with 
communication  and  infonnation  gathering. 

1.  Customer 


Metric 


Customer  Satisfaction  Index 


Since  the  individual  SYSCOMs  are  responsible  for  the  customer  survey  that 
eventually  is  used  as  input  into  the  Customer  Satisfaction  Index,  it  is  important  that  they 
understand  all  of  the  factors  that  affect  the  survey’s  validity.  Not  all  of  these  factors  can 
be  controlled.  There  are  two  issues  which  are  more  relevant  to  as  well  as  easier 
controlled  by  the  SYSCOMs.  The  issues  are:  1)  ensuring  that  the  survey  goes  out  to  the 
customers  every  year  and  2)  ensuring  that  the  DASN  ACQ  general  customer  satisfaction 
themes  are  covered.  These  two  issues,  out  of  all  the  ones  mentioned,  are  the  most 
important  in  receiving  relevant  data  for  the  metric. 

2.  People 


Metrics 

Employee  Survey 

Workforce  Stability 

Continuous  Learning 

DAWIA  Certification 

Acquisition  Professional  Community 

What  was  striking  about  the  Employee  Survey  is  what  was  learned  through  the 


interviews  with  the  SYSCOM  points  of  contact  for  the  DoN  Procurement  Metrics.  The 
response  rate  for  the  Employee  Survey  is  less  than  the  targeted.  One  SYSCOM  gave  an 
estimated  response  rate  of  about  9  percent  and  another  of  about  2.5  percent.  It  is  striking 
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to  find  that  out  of  four-hundred  surveys  sent  out,  ten  were  completed.  It  does  not  matter 
how  well  the  survey  is  written,  employee  satisfaction  cannot  be  effectively  measured 
with  such  low  response  rates. 

An  issue  with  the  Workforce  Stability  metric,  as  well  as  with  others,  is  the 
availability  of  the  infonnation.  It  would  seem  simple  enough  to  find  out  how  many 
employees  are  onboard  and  how  many  are  authorized,  but  this  is  not  the  case  for  every 
SYSCOM.  A  few  of  the  SYSCOMs  indicated  that  this  seemingly  undemanding  task  is 
extremely  time  consuming  for  them.  This  goes  back  to  the  issue  of  expending  effort  and 
resources  to  what  cost  and  what  benefit.  This  also  affects  the  objectiveness  of  this 
metric.  Is  the  data  accurate?  One  interviewee  indicated  this  about  the  military  portion  of 
the  Workforce  Stability  metric:  “[We]  just  do  not  have  the  military  data,  we  could 
estimate  it,  but  then  it  becomes  subjective.” 


3.  Process 


Metrics 

E2E  Metric 

P-Card  Delinquency 

DD1716 

Interest  Penalties 

Cycle  Time 

Consolidate  Service  Contract 

When  measuring  the  number  of  days  turnaround  time  ( DD1716  metric)  from 
receipt  to  disposition  there  may  be  some  room  for  gaming  the  system  with  respect  to 
when  exactly  the  Contract  Deficiency  Report  was  received  and  acted  upon.  The  different 
players  in  the  process  may  have  different  views  on  when  that  occurred  and  the 
individuals  who  control  the  input  into  the  NAFI  website  will  win  that  argument  and  could 
possibly  be  inputting  data  that  is  not  100  percent  accurate. 

The  same  sort  of  “gaming”  problem  exists  with  the  Cycle  Time  metric.  It  again,  is 
a  “days”  measure  and  it  relies  on  four  definitions  of  when  the  days  should  start  to  be 
counted  and  two  definitions  of  when  the  days  should  stop  being  counted.  The  Cycle  Time 
start  date  is  defined  as  when  the  earliest  of  the  four  events  listed  in  the  Metrics  Guide 
occurs  (date  of  Procurement  Planning  Conference  (PPC),  date  of  planning  meeting 
equivalent  to  a  PPC,  date  specialist  begins  work  on  procurement,  or  Procurement  Request 
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(PR)/Procurement  Initiation  Document  (PID)  receipt  date)  and  the  end  date  is  listed  as 
either  the  contract  award  date  or  for  work  in  process,  the  current  date  if  more  than  365 
days  have  elapsed  since  the  start  date.  This  may  seem  relatively  straightforward,  but 
there  appear  to  be  ways  to  manipulate  the  Cycle  Time  metric.  For  instance,  the  actual 
start  date  can  be  manipulated  by  the  individual  recording  it  to  start  later  in  order  to  appear 
to  have  a  shorter  cycle  time. 


4.  Financial 


Metrics 

Procurement  Direct/Indirect  Ratio 

Industry  Spend  Analysis — Competition 

Industry  Spend  Analysis — Small  Business 

Industry  Spend  Analysis — Commercial  Items — Actions 

Industry  Spend  Analysis — Commercial  Items — Dollars 

Industry  Spend  Analysis — Perfonnance  Based  Services — Actions 

Industry  Spend  Analysis — Perfonnance  Based  Services — Dollars 

What  is  difficult  with  regards  to  the  Procurement  Direct/Indirect  Ratio  metric,  as 
well  as  others,  is  that  the  Information  Technology  and  Human  Resources  system  that  is  in 
place  was  not  designed  to  generate  these  metrics.  In  one  of  the  interviews  it  was 
indicated  that  the  metrics  working  group  which  created  these  metrics  worked  with  IT  and 
HR  contacts  to  come  up  with  “friendly”  metrics  that  would  be  easier  to  produce. 
However,  several  of  the  interviewees  indicated  that  getting  the  necessary  data  was 
difficult  at  best  and  that  the  information  is  not  broken  down  into  the  categories  that  are 
required  by  the  metrics.  For  example,  the  salary  data  required  for  the  Procurement 
Direct/Indirect  Ratio  is  not  available  already  categorized  as  direct  or  as  procurement 
dollars.  Time  is  limited  and  the  interviewees  indicated  the  extremely  cumbersome  nature 
of  producing  the  required  metrics.  The  difficulty  in  collecting  this  metric  may  vary 
among  the  SYSCOMs  depending  on  what  resources  they  have  available  and  how  their 
data  is  kept. 

The  ability  to  obtain  the  HCA  total  procurement  salary  dollars,  for  the 
Procurement  Direct/Indirect  Ratio  metric,  was  described  as  a  complex  process.  In  the 
glossary  section  of  the  PMG,  it  states  that  “data  available  to  activities  vary  greatly,  no 
specific  way  of  calculating  procurement  salary  dollars  is  mandated.  These  approaches 
should  provide  a  reasonable  estimate  of  actual  labor  costs,  but  may  be  based  on  models 
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and/or  average  rates,  etc”  (PMG,  vl4.3,  p.  20).  This  means  that  each  HCA  could 
calculate  procurement  salary  dollars  in  a  different  manner.  If  the  HCAs  are  internally 
consistent  in  the  manner  they  collect  the  salary  data  and  their  method  of  estimation  then 
trends  can  be  analyzed.  They  can  be  analyzed  because  the  consistency  of  the  collection 
method  makes  the  data  points  comparable.  This  will  help  take  the  subjectivity  out  of  the 
total  procurement  salary  dollars  variable. 

Table  2  is  a  summary  of  the  findings  regarding  the  three  questions  described  as 
the  general  framework  for  evaluating  the  metrics.  There  are  varying  levels  of  objectivity 
and  completeness  as  previously  discussed  for  each  individual  metric.  For  purposes  of 
providing  a  summary  table,  “yes”  or  “no”  was  used  to  describe  whether  or  not  the  metric 
was  deemed  objective  or  complete.  The  issue  of  the  metric’s  link  to  value  was  covered 
only  for  the  seven  metrics  the  HCAs  collect.  As  stated  above  the  interviewees  are  of  the 
opinion  that  the  fourteen  metrics  collected  by  DASN  ACQ  are  currently  of  little  value  to 
the  HCAs.  The  remaining  seven  metrics  provide  some  value,  but  it  is  difficult  to  glean 
the  extent  of  this  value  because  none  of  the  interviewees  indicated  specifically  how  and 
when  they  are  able  to  use  them. 
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FOCUS 

AREA 

TIER  1  METRIC 

TIER  2  METRIC 

TIER  3 
METRIC 

TIER  4 
METRIC 

LINKED  TO 
STRATEGY 

OBJECTIVE 

COMPLETE 

RESPONSIVE 

LINKED  TO 
VALUE 

Customer 

Customer  Satisfaction  Index 

YES 

NO 

NO 

YES 

People 

Human  Capital  Index 

Employee  Survey 

YES 

NO 

NO 

NO 

Workforce  Stability 

YES 

YES 

NO 

Emma 

Qualifications 

Continous 

Learning 

YES 

YES 

YES 

DAWIA 

YES 

YES 

YES 

Acquisition 

Professional 

Community 

YES 

YES 

YES 

Process 

Process  Improvement  Index 

E2E  Metrics 

YES 

YES 

YES 

Efficiency 

Improvement 

P-Card 

Delinquency 

YES 

YES 

YES 

YES 

NO 

YES 

Interest 

Penalties 

YES 

YES 

YES 

Cycle  Time 

YES 

NO 

YES 

SOMEWHAT 

Consolidate  Services 
Contract 

YES 

YES 

YES 

Financial 

Contribution  Index 

Procurement 

YES 

NO 

YES 

SOMEWHAT 

Industry  Spend 
Analysis 

Competition 

YES 

YES 

YES 

Small  Business 

YES 

YES 

YES 

Commercial 

Items 

YES 

YES 

YES 

Dollars 

YES 

YES 

YES 

Performance 
Based  Services 
Contracting 

Actions 

YES 

YES 

YES 

Dollars 

YES 

YES 

YES 

Value 

Product  Unit  Costing  (PUC) 

Large  Contracts 

YES 

NO 

YES 

Simplified  Acquisition 
Procedures 

YES 

NO 

YES 

NO 

Table  2.  DoN  Procurement  Metrics  Evaluation  Summary 

Overall,  the  author’s  conclusion  is  that  the  metrics  are  not  providing  the  type  of 
value  that  performance  metrics  should  for  managers  in  an  organization.  Based  on  the 
interviews,  it  does  not  appear  that  the  metrics  are  being  used  to  make  any  decisions  about 
the  organizations  and  how  they  operate.  This  is  due  to  the  fact  that  fourteen  of  the 
twenty-one  metrics  are  not  seen,  so  they  are  in  essence  off  the  table,  and  the  other  seven 
have  varying  levels  of  value  to  the  HCAs,  if  any  at  all.  The  seven  metrics  that  the  HCAs 
report  on  (Table  3)  are  of  limited  value  for  two  primary  reasons.  First,  it  was  indicated  in 
the  interviews  that  estimations  of  variables  may  be  used  in  order  to  report  the  metric.  This 
not  only  creates  subjective  measurements  but  also  causes  inconsistency  over  time 
decreasing  the  validity  of  analyzing  trends.  Second,  while  annual  metrics  are  a  plus 
because  they  lessen  the  amount  of  resources  the  HCAs  have  to  expend,  they  are  not  of 
much  value  when  decision  makers  rotate  to  a  new  position  and  maybe  only  see  one  or 
two  sets  of  completed  metrics. 
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Focus  Area 

Metric 

Customer 

Customer  Satisfaction  Index 

People 

Employee  Survey 

Workforce  Stability 

Process 

Cycle  Time 

Financial 

Procurement  Direct/Indirect  Ratio 

Value 

PUC — Large  Contracts 

PUC — Simplified  Acquisition  Procedures 

Table  3.  HCA  Collected  Metrics 


5.  Analysis  not  Specific  to  One  Focus  Area 

Since  the  HCAs  collect  and  report  on  seven  of  the  DoN  Procurement  Metrics,  this 
focuses  the  SYSCOMs  on  seven  of  the  twenty-one  metrics.  This  being  said,  it  is 
important  to  note  that  not  having  to  collect  the  data  for  the  other  fourteen  metrics  is  a 
very  good  thing  in  the  HCAs  view.  They  would  just  like  feedback  on  them.  “All  the 
time  [we  have]  given  information,  we  have  never  received  feedback,”  was  a  comment 
during  one  of  the  interviews.  When  another  HCA  representative  was  asked  if  the  metrics 
help  them  in  any  way,  the  response  was,  “They  could,  but  [we  receive]  no  feedback.” 
Some  even  thought  it  valuable  to  see  the  outputs  of  the  other  SYSCOMs,  realizing  that  it 
is  difficult  to  compare  due  to  the  differing  nature  of  the  procurements.  Nonetheless,  a 
few  of  the  interviewees  were  still  curious.  One  interviewee  expressed  the  concern  that, 
“comparing  is  dangerous.” 

Feedback  was  mentioned  as  part  of  the  Cybernetic  Feedback  Model  discussed  in 
Chapter  II.  The  feedback  specific  to  the  individual  metrics  deals  with  the  variance  from 
the  standard  created  or  the  original  baseline.  Besides  the  Cybernetic  Feedback  Model 
type  of  feedback,  which  offers  insight  into  deviations  from  the  standard,  the  HCAs  desire, 
as  noted  above,  more  general  feedback  on  the  status  of  the  metrics  program.  The 
interviewees  asked  if  the  procurement  metrics  are  going  to  be  cheer  leaded  (have  a 
champion)  and  supported  by  DASN  ACQ.  If  DASN  ACQ  does  support  the  metrics  then 
the  interviewees  are  also  asking  what  resources  will  be  available  to  the  HCAs  in  order  to 
better  comply  with  and  complete  the  data  calls  in  a  timely  and  accurate  manner?  One 
quote  from  the  interviews  addresses  this  point:  “If  you  are  going  to  do  it  [use  the 
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metrics],  do  it  consistently  and  bring  it  to  attention,  that  will  bring  more  meaning  to  it.” 
The  idea  of  consistency  is  reflected  in  another  interview,  “stay  consistent  ...there  seems 
to  never  be  consistency.” 

Overall  the  feeling  was  that  the  metrics,  while  not  the  ones  each  specific 
SYSCOM  would  have  picked  to  explain  their  individual  success,  could  have  some  value 
if  they  got  the  appropriate  feedback  from  DASN  ACQ.  One  of  the  interviewees 
expressed  the  idea  that  DASN  ACQ  should  provide  feedback  in  the  following  manner, 
“We’re  seeing  a  downward  trend  [in  a  specific  area/metric],  and  you  may  want  to  look  at 
this,  this,  and  this.” 

Right  now  there  is  not  the  sense  that  DASN  ACQ  is  there  to  help  the  SYSCOMs. 
The  SYSCOMs  need  to  see  a  culture  of  “we’re  trying  to  help  you”  from  DASN  ACQ  to 
the  SYSCOMs.  Along  the  same  lines  one  interviewee  stated,  “[We]  need  it  as  a  system 
to  help  the  SYSCOMs,  not  the  SYSCOMs  fearing  they  would  get  in  trouble.  Show  how 
they  are  benefiting  the  SYSCOMs.” 

The  HCA  representatives  stated  that  the  fourteen  metrics  collected  by  DASN 
ACQ  are  of  little  or  no  use  to  the  HCAs  because  they  aren’t  getting  any  feedback  on 
them.  DASN  ACQ  has  indicated  that  part  of  this  reason  is  because  the  set  of  metrics  is 
not  a  finished  product  and  DASN  ACQ  feels  reluctant  to  put  out  all  of  the  results  of  their 
data  pulls  when  they  are  not  fully  convinced  that  the  metrics  are  the  correct  ones. 
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V.  CONCLUSION 


A.  SUMMARY 

The  Department  of  the  Navy  Procurement  Metrics  show  a  clear  link  to  the 
intended  strategy  of  the  five  Focus  Areas.  Each  metric  links  back  to  their  respective 
focus  area  and  is  aligned  with  Department  of  the  Navy  as  well  as  Department  of  Defense 
strategies  and  goals.  As  stated  in  the  literature  review,  when  a  metric  is  created,  the 
managers  creating  it  must  know  that  by  doing  so  they  are  telling  their  employees  what  is 
important.  The  simple  fact  that  something  is  being  measured  speaks  volumes  as  to  its 
importance. 

The  motivation  behind  each  one  of  the  metrics  is  important.  Each  metric  is  a 
piece  in  the  puzzle  that  when  put  together  should  have  the  ability  to  provide  an  accurate 
picture  of  the  health  of  the  Navy’s  procurement  function.  The  metrics  dashboard  has  the 
potential  to  be  a  great  management  by  exception  tool.  However,  this  tool  is  only 
effective  when  those  using  it  have  confidence  that  the  metrics  in  place  are  the  correct 
ones  linking  to  each  Focus  Area,  have  been  collected  and  reported  on  accurately,  are 
responsive  to  manager  actions  and  decisions,  provide  the  necessary  information  for 
managers  to  make  those  decisions,  and  have  appropriate  targets  set  for  each  metric.  The 
proper  use  of  the  metrics  dashboard  by  top  management  is  not  to  analyze  every  area,  but 
to  just  make  note  of  areas  that  are  not  “green”  and  ensure  that  people  execute  a  plan  of 
corrective  action  if  they  have  control  to  do  so. 

Each  of  the  metrics  has  had  a  preliminary  analysis  done  on  it  that  has  indicated 
that  all  of  them  align  with  the  intended  strategy.  The  analysis  continued  with  an 
evaluation  of  their  objectivity  and  completeness.  The  level  of  objectivity  and 
completeness  varied.  More  analysis  has  to  be  done  from  a  procurement  expert’s 
viewpoint  to  see  what  can  be  done  to  minimize  the  subjectivity  that  was  found  to  exist  in 
some  of  the  metrics,  specifically  the  customer  and  employee  surveys,  DD1716,  Cycle 
Time,  and  the  two  PUC  metrics.  Most  of  the  metrics  were  found  to  be  complete;  they 
fully  explained  what  it  is  they  set  out  to  measure,  but  some  are  not  (i.e.,  Customer 
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Satisfaction  Index  and  Employee  Survey).  One  of  those  which  is  not  complete  is 
Workforce  Stability.  It  is  clear  that  this  metric  needs  some  rework  as  mentioned  in 
section  B  of  Chapter  IV.  The  two  survey  metrics  were  evaluated  as  incomplete  mainly 
because  of  the  subjectivity  issues  listed  in  Chapter  IV  and  also  because  of  the  response 
rate.  An  employee  survey  that  has  a  2.5  percent  response  rate  is  highly  unlikely  to  give  a 
complete  picture  of  employee  satisfaction.  As  stated  before,  responsiveness  was  not 
evaluated  by  the  author. 

The  value  of  the  metrics  to  procurement  managers  was  also  analyzed.  Most  of  the 
conclusions  drawn  about  the  value  of  the  metrics  came  from  the  interviews  conducted 
with  the  HCA  representatives  from  six  of  the  ten  major  SYSCOMs.  It  is  unclear  as  to 
whether  other  personnel  within  the  HCAs  use  the  current  metrics  in  a  way  that  has  value 
to  them  or  have  ideas  for  certain  metrics  they  would  like  to  use.  Currently,  according  to 
the  interviewees,  fourteen  of  the  twenty-one  metrics  provide  little  value  to  the  HCAs 
because  in  their  view  there  is  no  feedback  on  the  metrics  that  are  produced  by  DASN 
ACQ.  The  remaining  seven  metrics  do  not  provide  infonnation  that  is  of  great  value  to 
the  HCAs  either.  This  is  likely  to  remain  the  case  until  a  trend  can  be  analyzed. 

The  idea  of  additional  metrics  is  not  one  that  the  HCA  representatives  embraced. 
This  goes  back  to  the  issues  of  value  provided  and  cost  to  provide  that  value.  One  quote 
from  an  interview  relays  this  point  fairly  succinctly,  “Any  additional  metrics  need  to  be 
evaluated  on  how  much  work  that  it  is  going  to  require  and  of  what  value  it  is.  Even 
collection  of  how  many  people  you  have  compared  to  end  strength  [is  cumbersome].” 

For  a  few  of  the  HCA  representatives,  the  opposite  idea,  getting  rid  of  the  metrics,  was 
something  they  wouldn’t  lose  any  sleep  over. 

B.  RECOMMENDATIONS 

If  the  DoN  Procurement  Metrics  are  to  be  used  and  be  successful,  they  need  a 
champion.  Partial  buy  in  will  only  prolong  the  period  of  indecision  on  the  metrics, 
engender  confusion  as  to  the  status  and  the  purpose  of  the  metrics,  and  potentially  waste 
valuable  time  and  resources  that  could  otherwise  be  used  more  effectively. 
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The  Procurement  Metrics  themselves  are  generally  good  measures  of  the  Navy’s 
Procurement  Function.  The  purpose  of  these  metrics  is  to  roll  up  data  from  all  of  the 
major  SYSCOMs  into  a  usable  diagnostic  tool.  There  is  no  perfect  set  of  metrics  that 
every  SYSCOM  would  have  agreed  on  as  telling  a  complete  and  accurate  picture  of  their 
procurement  activities.  A  recommendation  in  this  area  would  be  to  contact  the 
appropriate  person  at  each  HCA  to  find  out  which  of  the  metrics  they  (the  HCA)  deem 
valuable  for  her  or  his  SYSCOM.  It  could  be  as  simple  as  a  table  to  be  completed  like 
the  one  below.  The  table  does  not  list  all  metrics  or  SYSCOMs.  It  just  provides  an 
example  of  format.  The  actual  table  would  list  all  twenty-one  metrics  and  each  of  the  ten 
SYSCOMs,  who  would  mark  the  valuable  metrics: 


SYSCOM  1 

SYSCOM  2 

SYSCOM  3 

SYSCOM  4 

SYSCOM  5 

P-card  Delinquency 

X 

X 

X 

X 

E2E 

X 

X 

Cycle  Time 

X 

X 

Table  4. 

Sample  SYSCOM  Assessment  of] 

Metric  Value 

Feedback  has  to  be  initiated  by  DASN  ACQ  with  respect  to  the  metrics.  This 
paper  may  offer  DASN  ACQ  some  insight  into  the  metrics  themselves,  the  issues  that  the 
HCAs  have  with  specific  metrics,  as  well  as  the  metrics  initiative  in  general.  Right  now 
the  feedback  loop  is  not  connected,  it  is  open.  The  HCAs  need  a  sense  of  the  status  of  the 
metrics  and  feedback  on  not  only  the  fourteen  metrics  that  are  being  collected  by  DASN 
ACQ,  but  all  the  metrics. 

If  DASN  ACQ  decides  to  continue  with  the  DoN  Procurement  Metrics  initiative, 
then  the  HCAs  desire  the  necessary  resources  to  do  an  effective  job  at  collecting  and 
reporting  the  metrics.  Again,  the  level  of  effort  currently  expended  and  resources  already 
available  to  the  HCAs  most  likely  differ  from  SYSCOM  to  SYSCOM.  Two  interviewees 
from  separate  SYSCOMs  specifically  mentioned  that  larger  SYSCOMs  have  contractors 
to  help  with  surveys  while  they  did  not. 
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A  new  working  group  needs  to  be  established.  This  working  group  should  not 
only  be  comprised  of  the  HCA  representatives  for  the  metrics,  but  also  some  of  the 
managers  in  the  HCAs  who  would  specifically  be  using  and  benefiting  from  the  metrics. 

A  metric  by  metric  evaluation  can  be  done  in  much  the  same  way  as  this  paper  evaluated 
the  metrics.  This  evaluation  would  be  done  by  operators  and  managers  who  have  been  in 
the  procurement  field  and  who  know  the  ins  and  outs  of  the  processes.  This  means  that 
they  could  take  what  has  been  started  here  and  fine  tune  it  because  of  their  experience 
and  expertise  in  the  procurement  field. 

Several  issues  should  be  addressed  when  this  working  group  meets: 

1.  Reduce  the  subjectivity  of  the  indicated  metrics  ( Customer  Satisfaction  Index 
and  Employee  Survey )  to  an  acceptable  point. 

2.  Address  the  completeness  of  the  metrics  with  specific  emphasis  on  Workforce 
Stability. 

3.  Ascertain  the  actual  metric  responsiveness  to  manager  and  employee 
decisions. 

4.  Discuss  the  nature  of  the  feedback  that  will  be  supplied  to  the  HCAs. 

5.  Determine  what  additional  resources  are  available  to  the  HCAs  from  DASN 
ACQ  in  order  to  provide  the  most  accurate  and  useful  infonnation. 

6.  Discuss  the  definitions  of  the  variables  used  in  the  metrics,  specifically  those 
collected  by  the  HCAs,  in  order  to  reduce  any  guesswork. 

7.  Discuss  how  the  actual  data  needed  is  stored  and  what  has  to  be  done  to 
categorize  and  retrieve  that  data  to  put  it  in  a  usable  form  for  input  into  the  metric 
formulas. 

Addressing  these  points  by  DASN  ACQ  and  the  HCAs  is  likely  to  provide  very 
meaningful  information  on  the  next  step  for  the  metrics  initiative. 

The  results  of  this  paper  and  especially  the  outcome  of  this  new  working  group 

should  be  pivotal  in  deciding  the  future  of  the  DoN  Procurement  Metrics.  An  improved 

understanding  of  the  metrics  as  well  as  an  improved  set  of  metrics  can  only  be  achieved  if 
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this  issue  is  approached  openly  with  candid  discussion  of  the  issues  with  no  fear  of 
repercussions.  DASN  ACQ  needs  to  reassure  the  HCAs  that  they  are  there  to  help.  Buy 
in  at  the  top  should  have  a  positive  trickle  down  effect.  That  coupled  with  a  renewed 
sense  of  purpose  and  understanding  of  the  metrics  has  the  potential  to  transition  these 
metrics  into  the  Diagnostic  Control  System  it  intended  to  be. 
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